IRENAEUS
“If any one, however, advocating the cause of the Jews, do maintain that this new covenant consisted in the rearing of that temple which was built under Zerubbabel after the emigration to Babylon, and in the departure of the people from thence after the lapse of seventy years, let him know that the temple constructed of stones was indeed then rebuilt… (Irenaeus Against Heresies, Book IV chapter 34, 4)
======
The people (not all, of course) departed Babylon after 70 years of captivity to Babylon had elapsed. How long after it ended did they (some) return? They were able to leave because Cyrus had ended their captivity by Babylon and he had permitted them to return.